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Abstract. We derive critical noise levels for Gallager codes on asymmetric channels 
as a function of the input bias and the temperature. Using a statistical mechanics 
approach we study the space of codewords and the entropy in the various decoding 
regimes. We further discuss the relation of the convergence of the messsage passing 
algorithm with the endogeny property and complexity, characterizing solutions of 
recursive equations of distributions for cavity fields. 
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1. Introduction 

Error-correcting codes play a central role in modern communication. These are used 
to communicate reliably in noisy media such as satellite and mobile communication. 
Currently, there is a wide range of error-correcting schemes, ranging from the classic 
Reed-Solomon codes [1], used today in mass storage media, to the more recent turbo 
codes [2] and low-density parity-check (LDPC) codes [3], H] that have both shown near 
optimal performance. 

Error-correcting codes exploit the idea of introducing redundancy into the message. 
The extra 'redundant' bits are constructed in a way known to both sender and receiver 
by correlating the message bits. If the channel noise is not too high the receiver 
can successfully decode and retrieve the exact original message. To minimize the 
transmission costs the amount of redundancy must be as small as possible. This 
inevitably makes the code more prone to errors; any good error correcting scheme must 
minimize both the required redundancy and the error probability. It is not at all a 
priory clear what are the limits in this trade-off. It was in 1948 that Claude Shannon 
blazed the trail [5] and proved that good error- correcting codes with arbitrarily small 
error probabilities do exist as long as the amount of redundancy is not smaller than a 
certain level, the so-called channel capacity. 

However, Shannon's groundbreaking proof was not suggestive as to how to construct 
practical useful codes that reach the Shannon limit. In 1962 Gallager proposed the 
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family of the so-called low-density parity-check codes [31 H]. Although conceptually 
simple, this coding scheme was largely forgotten due to the computational limitations 
of the time. Currently, however, with the advent of the computer era, they are recognized 
as one the best schemes available. The LDPC are easy to construct, have a low 
complexity and perform near the Shannon limit [6]. 

Gallager codes were re-discovered by MacKay and Neal [7j at around the same time 
when Sourlas [8] showed that statistical physics can be used to estimate the performance 
of error correcting codes. These two events brought in a surge of activity as well 
as an influx of research ideas from physics to information theory and vice versa. In 
particular, from a physics viewpoint, error- correcting codes have been so far studied 
quite extensively. For example, low-density parity-check ones on binary symmetric 
channels [9], [TOl [HI [121 [13]) oii real- valued channels [HI [15], on irregular graphs [16] 
while more recently the error exponent was calculated in [T71 ITS] . Turbo codes have 
been studied in [191 120] • For a more complete review on the subject see [211 122] . Clearly, 
the bibliography of statistical physics of codes is largely biased towards the low-density 
parity-check ones: this is because the recently developed finite-connectivity techniques 
offer an ideal toolbox for the theoretical study of this field. Currently, the more recent 
developments in the physics of finitely connected systems allow one to extend previous 
results with algorithms that perform better [23l[2H[25]. Altough these algorithms reach 
the computational limits of today, maybe one day they will also become useful. 

Qualitatively speaking the emerging picture for Gallager codes is that for 
sufficiently small noise levels, decoding with a message-passing algorithm with a linear 
computational complexity in the block size is possible and the error-free state is the 
only stable state. For higher noise levels, one finds a transition to a regime where 
suboptimal states are created (marking the so-called spinodal or dynamical transition) 
and where the message passing algorithms fail to find the most probable solution. For 
higher noise levels, a second transition occurs (thermodynamic transition) where the 
error-free solution ceases to be dominant. This marks the upper theoretical bound for 
error-free communication. This means that block-wise maximum likelihood decoding, 
which is shown to be NP-complete [26], fails. 

In this paper we study Gallager codes on the family of binary asymmetric channels 
(BAG). The two extreme cases of this family include the binary symmetric channel 
(which has been the key actor in nearly all previous research) and the (fully asymmetric) 
Z-channel. The latter is used in communications through optical fibers. Within replica 
symmetry we calculate the location of the static and dynamic transitions. We also 
present phase diagrams describing where the frozen phase and clustered phases appear. 
As a reference point to test our theory we have used known results from information 
theory [27] and shown that it reproduces them with very good agreement. 

Our paper is organized as follows: In the following section we provide the model 
definitions for Gallager codes and the decoding process. In section [3] we set up 
the decoding problem in statistical mechanical terms. In section HI we derive the 
thermodynamic quantities for a simple limiting case (dense codes) while in section [5] 
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Figure 1. A graphical representation of the binary asymmetric channel: noise corrupts 
the different bits with a different probability. 



we compute the free energy of the binary asymmetric channeL In section [6] we discuss 
the failure of behef propagation (BP) while in section[7]we present results from a one-step 
replica symmetry-breaking scheme. We end this paper with a discussion in section |H1 

2. Model definitions 

2.1. Gallager codes 

The aim is to send a message reliably through a noisy medium. Hereby four processes 
are of importance: the generation of the message by the source, the encoding process, 
the noise and the decoding process. 

We consider a source which produces messages cr° G { — 1,1}^ with probability 

TV N 

PU^') = n Pini^-; b) = l[ h5^^ + (1 - 6)5,o,_i , (1) 
i=l i=l 

with h G [0, 1] the bias of the input signal. 

The message is sent from one point to another through a noisy channel. To 
communicate the message in an error-free way redundant bits are added to the message 
before it is sent through the channel (encoding process). The encoding process is defined 
by the map g : {-1, 1}^ {-1, 1}^^ : ^ a, with N < M. The elements of the 
image C of ^ are called the codewords. Shannon, in his original paper [5], showed that 
for a family of codes, having a completely random set of codewords, it is possible for 
— ^ oo to decode errorlessly with probability one as long as the code rate R = j^h{b), 
with h{b) = —blog2b — (1 — 6) log2(l — b) the binary entropy, is smaller or equal to 
the maximal admissible amount of information we can send through the channel. This 



is given by the so-called channel capacity C (see Appendix A for a computation of the 
channel capacity for the BAG). However, as Shannon's random encoding turns out to be 
inefficient for practical error correction a new encoding/decoding strategy was sought. 
Gallager, among others, proposed a scheme for introducing more structure in the set of 
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codewords 0,111 • In particular, he suggested the hnear space of codewords: 

C= jo- G {-l,l}^^|H*cr = l| , (2) 



with 



M 

(e*o-). = V^ = l,---,M-iV. (3) 

is the parity check matrix which is a sparse (M — A^) x M matrix with 



elements Hjj G {0, 1}. The symbol 



C1IC2 



denotes concatenation of two matrices. The 
matrix Ci is of dimension (M — A^) x and C2 is an invertible matrix of dimension 
(M — A^) X (M — A^). The elements of Ci and C2 lie in {0, 1}. In regular Gallager codes 
the parity check matrix is constructed such that there are K non-zero elements per row 
and C non-zero elements per column. In irregular codes the number of ones per row and 
per columns are drawn from a distribution. Counting the number of ones in this matrix 
provides for regular codes the relation R = [1 — C/K]h{h) which expresses the code rate 
in terms of the code parameters. The M — N equations implied by ^ are the parity- 
check equations. Using Gaussian elimination one can bring HI to a systematic form 
described by A = P 1m-n with P = "^Ci such that H = C2A and where 1^ is the 
£ X £ identity matrix. The matrices EI and A span the same space and are thus equivalent 
parity check matrices. We can now define the generator matrix G = such that due 
to the mod-2 arithmetic one obtains HG = [C1IC2] [IjvICa ^Ci]t = Ci + Ci = 0. With 
these definitions encoding is realized through cr = G*a^ . This implies that cji = cr° for 
1 = 1,. ..,N. 

Channel noise can be seen as a bit-flipping operation. The effect of noise can 
be presented as a transformation cr — > p = {u'lai, . . . , i/^^ctm), where G { — 1, 1}^ 
represents the channel true noise vector. The channel can be represented by the 
probability -Pchan(^''^|cr) of a true noise vector given the message. For the BAG, 
-Pchanli'^lcr) equals (flgureP, 

M 

(^.V.) , (4) 

i=l 

with 

^'chan(l^°k) = (1 - P)Sa -lSuO,l + - 1 ^i." ~ 1 + g^a.l^^O _i + (1 - g)5a,l^i.<',l • (5) 

The parameters p,q G [0,1] give the bit-flip probabilities of the channel. In (jll) 
we assumed that the channel is memoryless. For convenience we deflne the variable 
hi = p/ q E [0,1] such that the binary symmetric channel corresponds to k = 1 while 
for K = one obtains the fully asymmetric Z-channel, which is of interest for optical 
communication (with the two states representing the presence or absence of light in the 
channel) . 

The receiver at the other end of the channel uses a prescribed set of operations to 
extract the original message from the received word (decoding). After obtaining the bit 
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stream p the receiver is required to solve, using the aforementioned properties of the 
generator matrix, the equations EI * p = EI * i/. Among the solutions of these equations 
an estimate u for the true noise is obtained. Once this is found an estimate for the 
original message ar^ immediately follows. The estimates of the single bits are obtained 
by calculating the single bit marginals 

Pi = 5^Pdec(i^|p,H) = ^ Pdcc(l^lp). (6) 

The notation u\vi denotes the set of components of u excluding the i-th. The choice 
of Pdec(i^lp) determines the decoding process while v are the variables of this decoding 
process. At first sight it seems impossible to calculate these quantities as we need 
2*^~^ operations. However, LDPC codes owe their success in the existence of a belief- 
propagation algorithm [2H] (see also P, EH]), whose computational complexity scales 
linearly in the system size M, able to calculate the above marginals. This is achieved by 
interpreting the M — N parity check equations of ([2]) as a bipartite graph (the so-called 
Tanner graph) in which M variable nodes, associated to each z/j, are connected to M — 
check nodes associated to each of the constraints of ([2]). 

The performance of the code can be determined through a loss function [30]. If we 
take as loss function L(i/, = — X]f=i ^i^^-: which is the overlap between the true noise 
vector and the variables of the decoding process, the optimal estimator can be shown 
to be given by z>j = sign i^^. Pii^i) = sign(t'i) [31]. Thus we measure the performance 
through the order parameter p defined as 

I 

^ = M 5Z sign(z^iZ^°) • (7) 

i=l 

The brackets denote the average over ([6]) and the bar denotes the average over p and 
H. 

2.2. Decoding processes 

Without loss of generality we can represent the conditional probability Pacc{i^\p) through 

M 

idec(i^lp) = Ar(p) JJ exp {u,l3iH{) Sp^^i + exp {ui/3_iH_i) , (8) 
1=1 

distinguishing between different states for each received bit. The subindex corresponds 
with the value of the received bit pi. The normalization constant M{p) is independent 
of the decoding variables and will be left out. This will be important for the 
calculation of the entropy. The parameters Hi and also called the Nishimori 

parameters, determine the decoding scheme in the case of symbol-wise maximum a- 
posteriori probability (symbol-wise MAP). In symbol-wise MAP we want to choose 
the probability distribution Pdec{i^\p) such that p is maximal. Following [30] we can 
find these parameters by identifying ([8]) with the true posterior probability distribution 
-Ppost(^'lp) determined by the characteristics of the source and the channel noise. Using 
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Bayes' rule we obtain 

Ppost[l^i\pi) = ' , (9) 



and 



P{aM) = j^. • (10) 



One can easily write down the probabilities Pchani^^iWi) and P{pi\ai, z/j) from the channel 
description of figure [H For instance P{pi\ui,ai) = Y.a,p=±l^<'^,'^^p^,p^l'^,'^p■ ^or the a 
priory probability of codewords we have 

E,'5(H*.T = i)n,i,/'i„(<T,) 

since the first bits of the codeword are copies of the original message and all 
codewords must satisfy ([2]). From Ppost{^\p) = -Pdcc('^lp) we then find that the Nishimori 
parameters become 

/3- = l, g-.W^ilog '^^P'l'-"' . (13) 

2 qb 

In general we will consider decoding processes where f3i = = f3. When /3 — > oo we 
get block-wise MAP decoding. We remark that for unbiased channels symbol-wise and 
block-wise MAP decoders perform the same as symbol-wise and block-wise maximum 
likelihood decoders. 



3. Statistical mechanics for Gallager codes 
3.1. The partition function 

To begin the statistical mechanical analysis of the decoding process we define the 
equilibrium Boltzmann measure of candidate noise vectors given the parity check matrix, 
the true noise vector and the received bit stream: 

Pequil(i^|H, 1^', p) = ^ ] 5 [H * I. = e * Pdcc(l^lp) , (14) 

where Z[Il, u^, p) represents the partition function of our system: 

z(e, u\ p) = 5^ 5 [e * J/ = e * p^,,{i^\p) . (15) 

To find the behavior of p in ([7]), we calculate the typical value ft of the free energy 
/ = — ^ log 2 for M ^ cxD. Assuming self- averaging we can find ft by calculating the 
code- and noise-averaged free energy /, given by 

7= -^lim^-^5^P(H)5^Ppo,t(^-°,p|H)logZ(H,i.o,p). (16) 
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The hardcore restriction, which imposes that candidate noise vectors must satisfy the 
parity checks, can be written as 



5 [H * ly = e * = hm exp 



7— >oo 



7 X] ^iiJ2,-jK> {J-. 



:i7) 



with Jj,j,...j^ = ^l^j2--- ^jK a'^d 



1 if n£i = 1 for some i e {I, . . . , M - N} 



if otherwise 

The probabihty distribution of the tensor T follows from the statistics of H, namely 



(18) 



^m-^ n 



M 



c 



{K-l)\ 



j2,---,iif) 



1] 



1-C 



^=1 \{j2,-,jK);ji=l 



c 



(19) 



M. is the normalization constant, i.e. M. = e~ (^)*^. We have used the notation 
(ii! J25 ■ ■ ■ yjx) to denote the ordered set ji < j2 < ■ ■ ■ < Jk- The joint probability 
Ppost(^''^, p|IHI) in ( |T6|) does not factorize due to the asymmetry of the channel. It can be 
evaluated through 



Ppost(i^°,p|H) = ^Pprior(o-) 



(20) 



After making the gauge transformation i/j — > t'j^'f, we have the following partition 
function 

M 

1) 

0'lj2,---jA'> «=1 

modulo irrelevant multiplicative constants. The quenched fields hi are drawn from the 
distribution 



Z{{h,},m) = ^exp 



(5 (H * <T = 1) nil Pb(/ii, (^i) UZn+i P\ (^i' 

5 (H * ^ = 1) nf=i P6(a.) nSiv+i Pi (^0 



(22) 



with 



P6(/i,a) = (1 -g)W (or,l)5(/i-iJi(6)) +p(l -6)(5((T,-l)(5(/i + ifi(6)^ 

+(l-p)(l-6)5((T,-l)(5(/i-iJ„i(6)) +gM(a,l)(5(/i + /f_i(6)) , (23) 
and Pbi.o') = f dhph{h,a). 



3.2. Gauge transformation 



The gauge theory of disordered systems, pioneered by Nishimori [32], uses symmetry 
relations to derive a number of exact results. Of particular interest is the Nishimori 
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line on which one can compute exactly the internal energy and one can show that there 
are no replica symmetry breaking effects. For error correcting codes, using symbol-wise 
MAP decoding with {3 = 1 turns out to be equivalent to computing decoding observables 
on the Nishimori line. 

For an unbiased BSC we have pi{h, 1) = pi{h, —1). This model falls then in the 
category of channels characterized in [10]. Since the above distribution fl23|) fuUfils the 



conditions pb{—h, —a) 



-2h 



Pb{h, a) we can write the free energy in a more symmetric 



form as in [TU]. For any observable 0{h) we can write 

/ + 00 /• + 00 

p{hi, ai)0 (hi) = / dhi^p{hi,aiTi)e^^^^O{hiTi 
-oo Jo 



(24) 



with 



p{hi,ai) 



Pb{hi,ai) +pb{-hi, -cTi) 



2 cosh (hi) 

Making the transformations o"j — ajTj and also (tj, z/j 
we arrive at the more symmetric form 

/•oo M 



(25) 

{Tifii,Uifii), with 6(11* n) = 1, 



6 {M* n) exp 



■ M 



i=l 



log j 6 {M* u) exp 



M 



1=1 



(26) 



with n = ((JiTi, . . . , (TM'^'Af )• At /9 = 1 we can, using the techniques in [31], exploit 
this symmetry to prove that the thermodynamic state is replica symmetric. The energy 
e/3 = dpPf at = 1 equals 

AI 



e/3=i 



lldh,Vi{h,}) 



i=l 



M 



Jdap{a-b) E.Sim* a)YlLpa{a.^){h) 



h\cri,a 



The distribution p(a; h) is defined as 



PW b) = j^S{a -b)+ 6{a - 



(27) 



(28) 



The average {••■)h\a,a is over Pa{h\a). One can also prove that pp=i > pp which is 
equivalent to the statement that /3 = 1 corresponds with MPM decoding [33] . 



4. A simple solvable detour: The random codeword model 

Before we embark on the evaluation of the finitely connected case, we consider the 
simple limiting case of K,C oo. This limit, implying an infinite number of parity 
checks, is of course of small practical importance but nevertheless very educational as 
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Figure 2. The zero entropy lines, s(toi, to_i) = 0, for different rates i? at q = and 
p = 0.4 in the unbiased case of 6 = \. For rates R >C(p, q) the entropy is positive and 
decoding is not possible. Left: BSC. Right: Z-channel 



it already contains a wealth of information about the code's performance. It will also 
give us a first flavour about the effects of asymmetry in Gallager codes. In this limit 
it can be shown that the codewords a; G C, for an unbiased source, are sampled with a 
fiat probability, thus this model is coined the 'random codeword model' (RCM). These 
codewords determine the paramagnetic behavior of the system. Besides these, the model 
also contains the ferromagnetic state x^^'^ = a. We choose cr = (1, 1, ... , 1). We could 
say that this choice identifies Xj = ViV^^ which corresponds with the analysis done before. 
Below we follow the derivation as given in [10]. The energies of the codewords, after the 
gauge transformation Xj — > sign(hi)xi, are given by 



E ^ 



i=±i 



/=±i 



Hilnii 



with Ni = Eti^iMAHil) and = i;J:f=lH\hil\Hl\)(^^■ 
states, for a given mi and m_i, is equal to 



(29) 

The entropy of these 



s (mi, m_ 



(i?- l)log2 + 



g(mi) + 



P 



Q{m. 



(30) 



with Q{m) = — X]a=±i + -^"^)los[|(l + Am)]. The limit of maximum likelihood 
decoding is given by the noise levels [p*, q*) where s(mf , m^^) = 0, with (mf , m^^) the 
magnetizations of the ferromagnetic state: 



mi 



sign(ifi 



1 — q — p 



m 



sign (_?/"_ 



1 — q — p 



(31) 



1 — q + p 1 + q — p 

This zero entropy condition corresponds to R = T{p*, q*), with X the mutual information 
for an asymmetric channel, see equation ( ]A.3|) . We thus find the Shannon limit back, 
see also figure [2l In finite temperature decoding we restrict the energies ei and e_i by 
introducing the Lagrange parameters Pi and /5_i. The free energy /(/3i,/?_i) is defined 
through the Legendre transformation 

/(A, /5_i) = s(ei, e_i) - /?iei - /?_ie_i . (32) 
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We find that the entropy as a function of j3i and becomes zero when 
with 



l-q* +p* 



H 



1 -q* 



p 



l-q* ^ 
l-q 



p* 
't-p 



1 + q* 



p 



H 



1 —p* — q* 



p* 



H 



{l-qfi - p^( 



+ 



- q* 



p 



H 



p) 



PL, 



{1-pf-^+q^-^ 



(33) 



[1 - q)f^( + p 

The entropy can become negative as a result of having a partition sum dominated 
by atypical states. The number of these states becomes zero when M — > oo. This 
corresponds with an entropy crisis as found in the random energy model [3l], [35]. To 
avoid this we will introduce the spin glass phase corresponding with the ground states 
of the system. The spin glass state has a free energy fsc given by 

fsGi/3uP^i) = fpi/3(,Pi,), (34) 
with s{l3(,l3ti) = 0. The paramagnetic free energy fp is given by 

1 — q + p\ , 1 n TT { 1 — p + q 



-/p(/3i,/5_i)=/21og(2) 



log cosh f3iHi 



log cosh . 

(35) 



Comparing the free energies of the ferromagnetic, paramagnetic and spin glass state we 
find for /5i = (3-1 the phase diagram presented in figure O We remark that increasing 
the degree of asymmetry in the channel noise leads to a bigger ferromagnetic region. 
The ferromagnetic-spin glass phase transition is given by i? = X(p, q). The triple point 
lies at (/3i,/?-i) = (1,1). 

5. Free energy and saddle point equations 



In the more general case, the evaluation of the free energy (fT6i) and of the various 
thermodynamic properties can be done either with the replica [361 EZl [38] or the cavity 
method [391 SO], both of which have been shown to lead to identical results. Although 
the two methods differ in their philosophy, they can be seen as two complementary 
sides of the same coin, and together they can offer a more complete understanding of 
the physics of the system under study. We follow here the replica methodology and 



postpone our discussion on the cavity method for [Appendix B[ The free energy is of 
the form 



/ 



lim 



i(H*o- = i)nLK(<Ti)n 



rM 



5 (H * (7 = 1) pfe(ai) nfi^+i pi (a, 



-f{{<y^}) 



(36) 



with 



with the partition function Z {{hi} ,EI) given by fl2T|) . This expression, describing an 
average over parity check matrices and input codewords, can be dealt with using the 



^ „ A/ TV M 

-Pf{M) = j^ l[dh,l[p,{h,\a,) n P^-{hi\a,)hgZ{{hi},mp7) 

i=l i=l i=N+l 
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(p+q)/2 

Figure 3. The (T,(p+q)/2)-phase diagram for the random eodeword model with a 
rate R = 1/2 for different degrees of the channel asymmetry k = p/q. Solid lines 
indicate the thermodynamic transitions to the paramagnetic (PARA), spin glass (SG) 
or ferromagnetic (FERRO) phases . The dotted line represents the thermodynamic 
transition if freezing of the paramagnetic solution is ignored. The dashed line is the 
continuation of the PARA-SG line. 



replica method [39j. We replicate the cr-variables g times to cr = (a^, ■ ■ ■ , a^) and the p 
variables n times io u = [v^, ■ ■ ■ , z/""). The free energy per bit ( fT6l) is then given by an 
extremization problem: 

- 13] = lim^^extrpp$|p(i/,cr),P(i/,cr)|. (38) 



The function \& equals 



c 



K 



K 



K 



E n^'(-'.-')nMn-r,i)nMn-f.i 

C_ = \ \l=\ 



I/1,<Ti,I/2,<T2,---,Mk,<Tk ( = 1 



a=\ \l=\ 



/(a Q 
daqb{a) log < ^ Pb{(r^) ( (^(j^, cr)) exp 



I/,(T ( = 1 



h\a'- ,a 



with 



qb{a) 



C 



K 



C 



1 - — U(a - 6) + —6{a - -) . 



K 



m 



(40) 



The order parameters P {u, cr) and P (u, cr) are solutions of the self-consistent equations 



K-l 



K-l 



K-l 



a=l 



1=1 



(=1 



(41) 
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o C=3, K=6, spinodal 

* C=3, K=6, thermodynamic 




0.4 



Figure 4. Critical noise level lines in the (p, (7)-parameter space for symbol-wise 
MAP decoding and an unbiased source. Two Gallager codes of rate i? = 1/2 are 
compared. The channel capacity obtained from maximizing the mutual information 
over the input bias h versus the mutual information for an unbiased source (6 = 1/2) 
arc indistinguishable. 



P{u^<T) = / da M ^\ . (42) 

Inserting fj4T]) into P2|) produces a single self-consistent equation in terms of P{u,(t). 



5.1. Replica Symmetry 

For the joint distribution of the replicated spin variables P{v^ a) we now write 

P(i^,cr) = I da qb{a) P{(T\a) P{iy\cr,a) . (43) 



In order to take the limit n — 0, one has to make an assumption for the form of 
the distribution P{u\cr,a). The simplest such ansatz corresponds to replica symmetry, 
i.e. assuming that the a-replica indices with respect to the noise variables are inter- 
changable. More concretely we write 

dx 7r(x|cr, a) JJ^ Q{i^a\x) 

a=l 



Q{ I ) iPxh') ^^^^ 

2 cosh(/3x) 

for some 7r(x|cr,a) with J dx7r{x\(T,a) = 1. Using this ansatz we can convert the self- 
consistent equation of P{u,a) into one for the density 7r(x|cr,a) and one for P(cr|a), 
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cr a 



C-1 



TT X cr, a) 



c- 

n 



(45) 



We defined the messages 



^ c-i /x-i 
■U/3 /ij = h + — atanh | JJ^ tanh(/9x[ 



(46) 



To take the hmit g ^ we make the following assumptions on the cr-dependencies 



P{(T\a) 



7r[x\cr, a) 



dyv{y\a) l[Q{(r^\y) , 

J dz P(z\(T, a) 7r{x\a^, z, a) 

— — j— - / dz 9(z\a) P((T\z,a) n(x\a^ , z,a) 
P{cr\a) J 



(47) 



dyvivla) llQ{cr'^\y) 

C=i 



j dz e{z\a) 7r(x|a\ ;z, a) JJ Q {(t^\z) 

C=i 



(48) 

with J dz6{z\a) = 1 and / dxT[{x\a,a, z) = 1. The distribution ri{y\a) fuUfills the self- 
consistent equation 

/C-lK-l C-lK-1 
n n ) n n ^(^^^l^n Hy-^i ayn , yoia))) , (49) 
r=l Z=l r-=l /=1 

with yo{b) = ^ log (j^)- The distribution 9{z\a) turns out to be also a solution of the 
equation ( l49i) . The distributions 7r(x|cr, 2,a) are given through the equations 



/ 








S{z-Ui{{zl},yo{a))) 



X 



r,l 



JJ^ (ix[7r(x[|cr[, z[, a[) / dhp{h\a,a)6 {x — U/^{{xi} ,h)) 



with da qh{a) =T>f,a and dz 6{z\a) = T^Q a z and 



(50) 



(51) 
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Equations 0491) and (1501) are the main equations from which the various thermodynamic 
quantities in this section will be derived. We remark that the distribution 7r(x|(T, z,a) 
gives the distribution of cavity fields given the value of z on the corresponding link and 
the value of a and a on the corresponding site. Finally we have to average over a and 
z. Substitution of the ansatze (H3l) . ( Hil) . (H71) and (HHl) in the expression ( l39i) of the free 
energy leads to, after taking the limits — > and (7-^0 



— Irs 
with 



C 



^{K-l)] Ef ) ^F)ts' ({x,}) 



E 



(1) 



A4'i(K};/^) 



(52) 



E 



^ K K 

[gi{xi})]= Yl'^bai1^e,aiZi ^ P{{(Ji}\{zi})Y[dxi7r{xi\ai,Zi,ai)g{{xi}) 

1 = 1 ai,--,afr 1=1 



(53) 



^^''[fi{xn-,h)]= [v,{a)(]lv,a^^Ve,a^znY.Pa{a)]l J] P(K}k,K}) 

\r,l J a r=l a{-a'i^^^ 



( T \ T V 7'\ 



dhp{h\a,a)f{{x''i} ; h) 



and where we used the abbreviations 



(K) 



RS 



(1) 



RS 



- log ^1 + JJ tanh px}j + ^ log(2) , 

/ C / K-l 

- ^ log 5^ e''^ n 9 ( 1 + ^ n ^^^h 



(54) 

(55) 
(56) 



(57) 



r=l \ 1=1 

In the unbiased case 6 = |, we have the stable solution 9{z\a) = 6{z) and 7i{x\a, z, a) 
7r(a;|cr) with 



c-i 



TT X cr 



n E 



■)K-2 



r=l 



'1' ''"A'-l 



n 

1=1 



and 
— Irs 
with 



X / dhp{h\a,-)6{x - ui3{{x\} ,h)) 



|(K-1))eW ^Fg\{xi}) -E« AF«(K};/^) 



\C71,-,<TK 1 = 1 J 



(5^ 



(59) 



(60) 
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Figure 5. Thresholds of the noise variable g as a function of the bias for Z-channels 
with an encoding strategy {C^K) — (3,6). The thermodynamic and spinodal lines 
obtained through population dynamics are compared with the Shannon limit and the 
results obtained by belief propagation. The location of the minimum lies at 6 > i. 
The spinodal and thermodynamic lines are calculated for a set of 100 7r-populations 
each of 1000 x-fields. The BP-points are calculated on one graph instance with 10^ 
sites. 




(61) 



We see that for unbiased sources the formulas (|50l) and (|52l) are much simphfied to 
and fl59p . Generahzation of these formulas to irregular graphs is straightforward. 
We note that AF^^ and AF^^J correspond, in the framework of the cavity method |41j . 



to the free energy shifts due to link- and site-addition respectively. Equation fISS]) is also 
known as the 'density evolution' equation (while the equivalent (]B.16P of Appendix B 
which refers to a single graph instance is termed as the 'belief propagation' equation). 

We see that the state 7i{x\a, z, a) = 6{x — 00) is always a solution to fl50l) and gives 
p = 1. If the initial state lies in the basin of attraction of this solution, errorless decoding 
is possible. We will term this the ferromagnetic solution. The ferromagnetic state has 
a free energy equal to 



ffe 



- j daqb{a) j JJrfa[gb(a[) j Wdz\e{z\\al) 

r,l r,l 



C 



X 



E^»n E ^(K}k,K})W.k,a. (62) 
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= 1 




— 




1 


c 


K 


Ha 


He 


n 1 


He 




He 


5 


6 


0.13739(5) 


0.26436(3) 


0.35546(2) 


0.70400(5) 


0.28709(2) 


0.54800(1) 


3 


4 


0.16703(1) 


0.20959(1) 


0.45580(2) 


0.57591(4) 


0.35426(1) 


0.44167(1) 


4 


6 


0.11692(1) 


0.17245(1) 


0.30802(2) 


0.47132(5) 


0.24615(1) 


0.36373(2) 


3 


6 


0.08406(1) 


0.09972(1) 


0.23146(2) 


0.27880(1) 


0.17977(2) 


0.21329(2) 


4 


8 


0.07681(1) 


0.10717(1) 


0.20056(2) 


0.2905(1) 


0.16137(2) 


0.22635(2) 



Table 1. The spinodal (qd) and thermodynamic (qc) critical noise levels calculated 
within the replica symmetric ansatz at T = 1 and with an unbiased source. The 
variable k = p/q controls the amount of symmetry in the channel noise. The thresholds 
for the Z-channel are calculated with k ~ 0(10"*). 



From equations (!59|) . (162|) we see that for k = 0, the free energy becomes — oo. To avoid 
these infinities we will solve the problem for k ^ and look at quantities that are finite 
for ft — > 0. 
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Figure 6. The {T, {p + q)/2)-phase diagram for the spinodal transition lines. At 
these lines the 7r(a;|cr) = S{x) state lies at the boundary of the ferromagnetic basin of 
attraction. The lines are calculated within the replica symmetric approximation for a 
{C,K) = (3,6) regular Gallager code with an unbiased source. The variable n — p/q 
controls the amount of asymmetry in the channel noise. The Nishimori line, T = 1. is 
visualized. 

We solve the coupled set of equations ( l49l) and ( l50l) using a methodology similar 
qto 'population dynamics' [IQ]. We first derive the stationary distribution for the 
density (H9|l describing a population of ?/-fields . . . , ?/„}. We remark that since 
6{z\a) = r]{z\a) we need not to update separately a population of 2;-fields. For every 
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l/-field yi we associate a vr*^*^ -population of x-fields, namely yi — > {xf'*, . . . , Xm }• A 
stationary solution for the population of populations of x-fields is found through the 
following algorithm: 

(i) We select {C — 1){K — 1) fields from the y-population: {yi) with £ G iS and S the 
set of chosen indices S = {ii, . . . ,i{c~i)iK~i)}- 

(ii) We calculate a new 2;-field according to its update rule: = Ui(^{yi}£(zs,yo{a)j 

(iii) We use the (C — 1){K — 1) populations of x-fields indexed by S to calculate a new 
population of x-fields with the update rule Xy, = M/j^jx^j^g^, fij 

(iv) We select at random an index j G [0,n] and replace the j-th member of the y- 
population by z^^ and the j-th member of the x-population by x^. 

We start with the initial distribution 7r(x|(T, z, a) = 6{x). This corresponds to a state 
with no a priory knowledge on the message cr°. At low temperature this distribution 
converges to the ferromagnetic state. Increasing the noise at a constant temperature 
we find that at some critical noise level {pd, qd) a second solution appears (suboptimal 
solution) with p < 1. We remark that below the (p^, gd)-threshold, the ferromagnetic 
state is the only stable state for all initial conditions. From an algorithmic point of view 
{pd, qd) is the threshold to successful decoding with the behef propagation algorithm. 
The thermodynamic transition is determined by the point {pc, qc) where the free energy 
of the suboptimal solution becomes lower then the free energy of the ferromagnetic 
solution. At T = 1 this determines the limit for maximum likelihood decoding. In 
figure m we see that when we increase C and keep the rate constant, the limit for 
maximum likelihood decoding increases but the limit for belief propagation decoding 
decreases. Indeed, we found that in the RCM the paramagnetic state is always stable. 
We find that, in contrary to the cases of symmetric channels, the bias b in the input 
signal influences the decoding process. However, comparing the critical noise levels 
{ps, qs), above which errorless decoding is impossible for all type of encoding processes, 
between an optimal biased source and an unbiased source, see figure lAlt we see that 
bias has few influence. In figure [5] we show, for a Z-channel, how these thresholds 
get influenced by the bias. For a BSC the minimum lies at 6 = |, whereas for a Z- 
channel the minimum channel noise q lies at a point 6 > |. We also compared our 
results obtained through population dynamics with a specific application of the belief 
propagation algorithm on a specific graph instance. The results of both methods match 
very well. In table [T] we present the spinodal and thermodynamic critical values for 
different regular codes and for various degrees of symmetry. These results are consistent 
with values found in [221 E] and in good agreement with those of [27] . In figure [6] we 
present the spinodal transition lines for different values of the inverse temperature f3 and 
the parameter k. We find re-entrance effects below the Nishimori line. We find these 
re-entrance effects also in the thermodynamic lines. In figure [7] we plot the entropy 
s = —df/dT, with / determined through equation (!59l) . as a function of the channel 
parameters. We see that it becomes negative at the spinodal noise level {pd,qd)- The 
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entropy at the Nishimori temperature has a special meaning as it is the average entropy 
of the transmitted message once the received message is known. This is therefore the 
theoretical upper limit irrespectively of the decoding dynamics. The energy at T = 1, 
see equation (l27|l . equals the ferromagnetic energy (162|) . From this it follows that the 
entropy at T = 1 becomes greater than zero at the critical noise level (pc, Qc)- Performing 
a large {C,K) expansion of fl58|) and fl59l) . as done for the BSC in [10], we get for the 
critical noise levels, taking n = 

1 ( d 



- constant. 



21og(2) 



[l-R) 



-1 



\ 



K 



with l{Kqf\qf^] 



l + {n-l)qi'^) (l - - l)g(°)) 

= R. The function v is given by 
1-gf {K + l)f 



+ 0({v{qi'^)) 



K 



(63) 



(0) 



i)g^ 



(0) 



X max < —2{k — \)q^ 



(0) 
c ; 



(K 



(l-(K-l)g(°))(l + (K-l)g, 



(0)^ 



(64) 



From dM]) we find for a (3,6)-code when k = 1, = 0.103968 and k = 0, = 0.284897. 
For a (3,4)-code we find when k, = 1, pc = 0.213414 and when k = 0, pc = 0.579815. 



5.2. The entropy crisis 

From figure [7] we have learned that the entropy can indeed become negative for the 
asymmetric channel. We also found re-entrance effects in the thermodynamic transition 
lines. This indicates that something is missing in our solution. In the SK-model |39j . 
the negative entropy in the ground state is an indication that the replica symmetric 
formalism is incorrect. In general p-spin models, see [12], we have at a temperature 
a transition from a paramagnetic phase to a one step replica symmetry breaking phase 
and at a temperature Tk < Td an entropy crisis corresponding with the vanishing of 
the configurational entropy. Here, though, because of the infinitely strong interactions, 
the first phase transition will not appear [13]. We will have an entropy crisis just like it 
occurs in the RCM. Because we are interested in the typical behavior of the system we 
should define the typical free energy /f(/3), as 



m 



(65) 



l^s{^) if SRS>^ 

fj,sm if sns<0 ' 

where /?/ is the inverse temperature at which the system freezes in the lowest energy 
paramagnetic configuration, i.e. s{l3f) = 0. Because of the hard constraints, we have 
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Figure 7. The entropy s as a function (p + g)/2 of the stationary solution of 
the density evolution equation ((58|) . starting from ■K(x\a) = 5{x). The lines are 
calculated at the Nishimori temperature for a (3,6)-regular code. From left to right: 
K = {0,0.01,0.1,0.25,^,1}. The sub-optimal ferromagnetic solution at a certain 
critical noise level emerges with a negative entropy. This solution becomes the 
thermodynamic one when the entropy becomes positive. 
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Figure 8. The thermodynamic (T,(p+q)/2)-phase diagram in the frozen ansatz for 
a regular {C,K) = (3,6) code and unbiased source. A FERRO, PARA and SG phase 
occur. 

indeed that following frozen ansatz, see |10j . 

(„ n/m \ In/m m \ 

/ dxTi^{x\(T, a) J] J] 11 ^ K". ^1 . (66) 

7=1 ) \j=la=l ) 
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Figure 9. The endogeny parameter de as a function of the temperature T for a regular 
{C,K) = (3,6) Gallager code. Left: binary symmetric channel, right: Z-channel. 
Lines represent the endogeny parameter calculated through population dynamics (the 
de presented for the Z-channcl is for a = —1). For T such that d^ > no meaningful 
solution exists to the decoding equations. Markers indicate the values de calculated 
for the log-likclihood ratios of the belief propagation fields at different times steps. 

with m G [0,1], fuUfills the selfconsitent equations P2l) . Using this ansatz in the 
self-consistent equations ( l42l) and the free energy expression ( l39l) . we find back the 
rephca symmetric equations (!50!) and (152|) with /3 — > /3m. The extremization condition 
= corresponds to the zero entropy condition, which for m G [0, 1] can only be 
fuUfilled when (3 > (3f. When (3 < /?/ we have m = 1, because there the free energy 
is indeed maximal. This corresponds with the frozen scenario of fl65|) . We will call the 
phase where the entropy is zero and p < 1 the spin glass phase and the phase where 
s > and p < 1 the paramagnetic phase. In the spin glass phase the thermodynamic 
average is dominated by a subexponential amount (in the system size) of codewords 
whereas in the paramagnetic phase the average is dominated by an exponential amount 
of codewords. In figure M we plot the full thermodynamic phase diagram of the system 
in the space of (T, \{p + q)) for a regular (C, K) = (3, 6) code with an unbiased source 
and three different levels of symmetry in the channel noise. The re-entrance effects have 
disappeared because of the frozen ansatz. 

6. Non-convergence regions of belief propagation and endogeny 

Although the freezing scenario presented above seems to explain the thermodynamical 
phase diagram completely, the dynamics of the system can be disturbed by a clustering 
of the phase space. To investigate this more closely we will consider a system composed 
of two copies of the original dynamic variables. These are embedded on the same graph 
and interact with the same quenched fields. In this setting, convergence of the recursive 
decoding equations can be quantified through the resulting statistics of the joint system. 

We remark that the convergence of the belief propagation equations can be seen 
as one example of a general class of problems in which one is interested in the 
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stationary density that solves a distributional fixed-point problem. This problem and 
its applications to various fields has been studied in a rigorous way by Aldous and 
Bandyopadhyay [S] from the viewpoint of theoretical statistics. This link between the 
two fields has been observed in [15]. 

In the case of a two-replica system we define a partition function of a form similar 
to (EI): 



Z {{hi} , H) = ^ exp j 7 ^ Tn,i2,-,iK {^h^i2 ' ' ' + /^n/^i2 " " " /^i^ ) 

X exp ^ {hiVi + /ij/ii + -ff.ui^ifii + jf,fii + juT^i)j , (67) 

with 7 oo. Analagously as in section [5] we find for the unbiased case, the following 
order parameter equations 



(i/l,/il,(Ti),---,(jy^_l,/x^-_l,CT^-_l) 

K-l \ / K-l 



X 



2K-2 



^ ^ ' l[P{iyi, filial) 



1=1 



1=1 



1=1 



c-i 



P{fi,iy\(r)) exp(/3/iEaK + /i")) 



h\(7 



C 



E^,. {(Pit^, ^k) exp i^h (i^" + /X")) 



(6^ 



(69) 



h\cT 



We introduce the replica symmetric ansatz 

P(.,M|a) = j dx^ ).(x( V) (4eosh(/3xa))cosh(/jx(^)))" " 
We remark that a field coupling the /x and variables is not needed because in fl69l) 
the quenched field does not couple ^ and variables. Substitution of this ansatz in the 
above self-consistent equations leads to 



7r(x(^U(2V) = / n 



c-i 



r=l 



1=1 



x6 



1' '""K-l 



5 



(71) 



Now we check whether the distribution 7ro(xi, X2I0") = 7ro(xi|(7)7ro(a;2|cr), with 7ro{x\a) the 
solution to converges to 7r(xi, X2|a) = 7ro(xi)5 (xi — X2) (note that all computations 
are done within RS). We introduce the quantity 

f dXidX2TT{Xi,X2\o')\Xi — X2\ 



deiTT{Xi,X2\a)) 



J dXidX2TT{Xi, X2\0') ^^^iili-^ 



(72) 



We remark that dg = corresponds in the two replica formalism to g — = and 
hence corresponds with some sort of spin glass behavior in the same sense as in the SK 
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Figure 10. Full (T, (p + (7)/2)-phase diagram for a (3,6)-encoding scheme over a Z- 
channcl. The solid lines indicate thermodynamic phase transitions and the dotted 
lines represent spinodal transitions. The dashed line determines the thermodynamic 
transition in the replica symmetric approximation. The vertical line determines the 
same transition in the frozen ansatz. The dashed-dotted endogeny line bounds the 
region below which the BP algorithm stops converging. 

model. This two replica formalism is, in certain models, proven to be equivalent with 
the endogenous property, see [H]. The failure of the endogenous property has serious 
consequences on the convergence of the BP equations. We define 

M 
i=l 

with hf^ the log-likelihood ratio on site i on the t*^ time step of the BP algorithm flB.16p . 
As long as \im.t-,oode{'^^Bp{x''^\x'^'^^)) = 0, the BP equations converge. In figure [9] we 
compare the parameter de of both formalism and we find that indeed c/g > when the 
BP equations stop converging. We call the line marking the transition from = to 
> the endogeny line. In figures [10] and [TT] we present this line respectively for a 
Z-channel and a BSC, together with the different thermodynamic and spinodal lines. 
In table [2] we give a summary of the various regions of the phase diagrams in figures fTOl 
and [m Performing a high connectivity expansion, like is done in [10] we find that the 
endogeny parameter of the paramagnetic solution is zero. 

7. IRSB ansatz 

From the results of the previous section we know that replica symmetry fails below a 
certain temperature. The non-convergence of the belief propagation equations below 
a certain temperature reveals that the amount of solutions of the belief propagation 
equations (IB. 161) scales exponentially with the system size. This can be solved using 
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Figure 11. The same lines as presented in figure [10] for a BSC and a (3,6)-encoding 
scheme. Two additional points are added where a full IRSB calculation is performed. 





convergence of BP 


free energies 


F 


yes 


\ 


FP 


yes 


/ferro < /para 


P 


yes 


/ferro > /para 


SGI 


yes 


/ferro > /SG 


SGII 


no 


/ferro > /sG 


FSGI 


yes 


/ferro < /sG 


FSGII 


no 


/ferro < /sG 



Table 2. The different labels used in figures [TUl and [TTl The convergence of the BP 
algorithm is indicated. The free energies of the stable states are compared. 



replica symmetry breaking, which correspond to a more advanced algorithm. In 
optimization problems, using insights of IRSB-effects, practical algorithms have been 
found (see [46]). To count the number of the solutions of ( 1B.16I1 . we introduce a Lagrange 
parameter conjugate to the free energy of these solutions. We have a generalized free 
energy $ corresponding with 

-11^= log exp {-II fa) , (74) 

with a a sum over pure states. By pure states we are referring to independent ergodic 
components in our system. If we call P„ = ^xp {—fifa) / (Xlo^^P i.~l^fa)) with /„ the 
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free energy of state a we see that 

-5^P,log(Pj=/i(*-/), (75) 

a 

with 

-f3f=Y,PJo.. (76) 

a 

These quantities can be calculated through the following ansatz (see 



r ^ / /• exp (Bx \ 

P{v\cT, a) = J dnVmsB (7r|cr, «) 11 I j dx7i{x) ^2 cosh (/3a:))"' I ' ^^^^ 

for some functional 'PiRSB[vr|cr, a] with J ■DttPirsbIttIct, a] = 1. Replicas here are only 
interchangeable within the group a = 1, . . . , ^ to which they belong. Spin variables 
carry two indices denoting the group a and replica 7 within the group. This one-step 
replica symmetry breaking has been considered for the binary symmetric channel in |23j . 
Substituting this ansatz into the self-consistent equations results in, using 



c-i 



^iRSB (7r|cr,a) = JJ 



/ E.,,.,.,., S {a Ui <Tu 1) P{^i\ai) 



JJrf7r['PiRSB « I CT;, a/) j j dhp{h\a^,a)SF 7r(x) - W(x; {<} , /i) 



(78) 

where (5i;'[.^(x)] denotes a functional delta distribution in the sense that Q[f] = 
J d^Q[^]6F[i{x) — f{x)]. We also introduced the distribution U{x; {vrj} , h), equal to: 

„ C-lK-l 

U{x- {<} ,h)= WW dx\7,\{x\) exp (-/iAF) 5(x - up ({x[} ,/.)). (79) 

r=l 1=1 

In f lTU]) AF is given by 

AF = - i log exp (/3/ir) jf ^ (^1 + r H tanh(/5x[) j j . (80) 

which in terms of the cavity terminology equals the free energy shift due to iteration. 
When we focus from now on on the unbiased case we get a somewhat simpler expression: 



r=l \ai,(T2,---,crK-l 1=1 



IRSB (t^IWi) 



X J dh p{h\a) Sf [vr(x) - U{x; {<} , h)] . (81) 

In principle this equation can be solved with the iterative scheme of population dynamics 
[ID]. Substitution of ( 1771) and (1781) in (l39l) produces an expression for the generalized 



E' 
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- ^msB = {K - 1)^ eSSb [a$ 
with the averages 



IRSB 



(1) 



(1) 
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(K};/^) , (82) 



c 



<SB [giun ■^h)] = -J2\Il E 



X 



The generahzed free energy shifts A$^^gg and A$^^gg are given by: 



A* 



A$ 



IRSB ~ 


-i,o 


(1) 

IRSB ~ 





K 



Y\_dxi 7ri{xi\ai) exp -/iAFj 



RS 



C 



r T / r \ r\ 



exp 



1=1 r=l 

The free energy follows from /irsb(/u) = d{fi^)/d^: 



— flRSB 

with 



^ J IRSB 
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(1) 
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^ J IRSB 
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AFif exp 
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Jlti dxi7ii{xi\ai) ^ 


exp 







I 
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-/^a4s^^ 
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Combining (182|) and (187|) produces finally the complexity: 
S(/irsb) = yu/iKSB - At$(At) , 



(83) 



(85) 
(86) 

(87) 

(88) 
(89) 

(90) 



as a function of the free energy. An alternative way to derive the above is based on 



the cavity method (see Appendix B) which shows that the complexity corresponds to 
the entropy of the number of solutions to the cavity equations with free energy density 

/iRSB- 



7.1. A special case: the frozen- ansatz 

The general scheme described by 0781) allows one to retrieve the solutions of the frozen 
ansatz fl66l) . This can be done by considering solutions of the form 

P[7r|a] = j da Q{a\(T) 5 p(^{x) — a5 {x — oo) — {1 — a)5 {x + oo)^ . (91) 
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Figure 12. The free energy / — efcrro (solid line) and the energy difference e — efcrro 
(dashed line) as a function of the inverse temperature (3 for a regular (C, K) = (3, 6) 
Gallager code. At the point / = e the entropy is zero. This point determines 
the thermodynamic value of / at the frozen transition. These graphs can also be 
interpreted in the frozen ansatz ([9T|) in IRSB, with the identification (|93|) . Left: k = 1 
(binary symmetric channel) with p = {0.09,0.1,0.11} from top to bottom. Right: 
K = (Z-channel) with q = {0.25,0.2788,0.29} from top to bottom. 




Figure 13. The entropy s (complexity S) of a regular (C, K) = (3, 6) Gallager code 
as a function of the energy difference brs — eforro (free energy difference /irsb — Cferro)- 
Left: binary symmetric channel, right: Z-channel. Above the thermodynamic 
transition (upper line) the complexity S(/fcrro) is positive implying that there are 
exponentially many codewords with the same free energy and as a result decoding 
fails. 



This solution can be interpreted as having on each site a probabihty a to have a state 
with V = 1. The distribution Q{a\(y) corresponds to site averages. In the case where 
C is even, it is clear that (1911) is a solution of the IRSB self-consistent equations (1781) . 
For odd C the reweighting factor e~^^^ in (178|) makes sure that the zero fields do not 
appear. Hence (1911) is also a solution of (1781) when C is odd. Substitution of (1911) in (ITHl) 
gives the following self-consistent equation for the distribution Q{a) 

r=l \ cri,CT2,--,o-K-i 1=1 I 
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a — 



exp (/i/i) 



c-i 

n 



5+ — exp (yu/i) Y\ 



r=l 
C-1 



r=l 

C-1 



K-l n 

E E IKi-o;:) n < 

n=0 niseven (/i, ■■■,/„) i=l lf(li,---,ln) 
K~l n 

E E n(i-<) n 

n=0 niseven (ii, i=l l^{h,''',l-n) 



exp 



r=l 



(92) 



E E 11(1-4) n 

n=0 nisodd (ii, ■■■,;„) i=l /^(/i, ■■■,/„) 

These equations turn out to be equivalent to the RS equations ( !58l) . we found before. 
This can be seen by substituting a = exp (fix) / (2 cosh (/3a;)). Then setting /i = /5 we 
obtain the following identities between RS and the present ansatz (l9Ti) : 

ElRSB = Srs , $1RSB = /rS , /iRSB = crs • (93) 

We have thus returned to replica symmetry with the IRSB free energy playing the role 
of the RS energy. From figure [12] we see that indeed fRsi^iRss) reaches its maximum 
at srs = (J^ijisB = 0). From this it follows that we can maximize the free energy for 
P < Pg with m = 1, while for (3 > (3gWe obtain m = Pg/p. The relationships (|93|1 can 
easily be interpreted through: 



- Ai$iRSB(/i) = log I ^ exp(-/i/< 

\states a 

= 'og( E 

\states 



exp [-jJ^ec 



-pmfRsipm) 



(94) 



where we used that Sa = 0. In figure [13] we see that above the thermodynamical 
noise levels {pc, Qc), the number of codewords with an energy equal to the ferromagnetic 
codeword scales exponentially with the system size. These figures can be compared with 
the figures [2] Just like in the RCM we can interpret the thermodynamical transition 
{Pc, Qc) at T = 1 as the theoretical upper limit for succesful decoding with (3,6)-Gallager 
codes. 



7.2. The more general case: the complete IRSB 

Finally we consider the solution of the IRSB equations ( IHTl) and ( 182|) . Numerically the 
IRSB approach is prone to many errors coming from the small sizes of the distributions 
(we used 1000 distributions of each 1000 fields). We also emphasize that the IRSB 
replica and cavity method may contain many non controllable approximations. This 
is especially true for the complexity, see [H] and [19]. With this in mind we try to 
interpret the result presented in figure [HI which has been calculated for parameter 
values corresponding to the marked points of the phase diagram in figure [11] At the point 
marked with a cross we find a zero complexity for all values of m. At the dotted marker 
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Figure 14. The complexity E and the entropy s as a function of the rcphca symmetry 
breaking parameter m for a BSC and a (3,6)-code at p = 0.09 and T = which 
corresponds to the marked point (•) in the phase diagram of figure fTTl 



we find a positive complexity for some values of m. From the thermodynamical relation 
between m and S, we know that S must decrease as a function of m. Eliminating the 
branches where the complexity increases as a function of m we find the results in figure 
[HI We find a regime with a positive complexity and a negative entropy. This means 
that there is an exponential number of solutions to the belief propagation equations 
and thus the belief propagation algorithm does no longer converge. We also remark 
that the fact that these solutions have a negative entropy is consistent with the freezing 
picture we found in section 17.11 It would be interesting to look for the change of the 
dynamic thresholds between the replica symmetric and IRSB algorithms. In order to 
exclude finite size effects, we would need larger system sizes to determine accurately 
these thresholds. 

8. Discussion 

In this paper we study the decoding properties of LDPC-codes on a binary asymmetric 
channel, using tools from statistical mechanics on finitely connected systems. As a 
result of the channel asymmetry the microscopic Boltzmann distribution for the channel 
noise inherits an explicit dependence on the received message. This results in a set of 
recursive equations for two types of cavity fields. We determine the decoding thresholds 
for message passing algorithms as a function of the important parameters, e.g. the 
asymmetry, the bias and the temperature. Calculating the entropy we find the upper 
bound to any decoding scheme. 

For dense codes we retrieve the random codeword model. The thermodynamic 
averages are characterized by the existence of a ferromagnetic, spin glass and 
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paramagnetic phase. The ferromagnetic region increases with increasing asymmetry 
in the channel noise. Because the paramagnetic solution is always stable, the message 
passing algorithms fail to decode correctly the received message at all noise levels. 

For low-density codes the emerging picture in the temperature-noise phase diagram 
is that for high temperatures we find two solutions for the cavity distributions. For 
these temperatures and low noise levels there is a ferromagnetic phase indicating 
succesful decoding. Increasing the noise level to a certain threshold the appearance of a 
paramagnetic solution distorts the decoding process. As the temperature is lowered this 
paramagnetic solution freezes into a zero-entropy solution, representing a subexponential 
number of codewords. Lowering the temperature even further the decoding dynamics 
of the system is distorted by an exponential number of metastable states. We discuss 
this failure in terms of the endogeny property of the recursive equations for the cavity 
fields. 
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Appendix A. The capacity of the binary asymmetric channel 

Shannon's famous channel coding theorem states that error free communication can 
be possible as long as the rate R = jjh{b) is kept below a certain critical value C, 
the channel capacity. We use the abbreviation h{b) for the binary entropy, h{t) = 
— tlog2t — (1 — t)log2(l — t). We here calculate the channel capacity for the binary 
asymmetric channel. It is defined as 

C = maxJ(X,r), (A.l) 

p(X) 

where X = {xi,X2, ■ ■ ■ ,xl} is the set of possible inputs to the channel and Y = 
{Vi, 1/25 ■ ■ ■ 5 2/1,} the set of possible outputs. The average mutual information 

X{X,Y)=H{Y)-H{Y\X), (A.2) 

written in terms of the marginal- and conditional entropy represents the amount of 
information carried by the channel for a given noise probability. Thus C provides the 
maximum admissible amount of information carried by the channel. In the case of a 
binary alphabet with L = L = 2 and with p{X) = bSx,xi + (1 — b)6x,x2 we find that for 
the binary asymmetric channel, see figure [H the mutual information is given by 



(A.4) 



2 

The maximum of T{p, q) with respect to b is attained at 
^ p(exp[F(p,g)] + l)-l 

{q + p-l){l+exp[F{p,q)]) ' 

with F{p, q) = log 2, provided that 6^ 7^p/(p + ? — !)• Indeed, we see in figure 

(lAll) that at constant rate the channel noise gets a maximum at some 6 7^ i. If on 
the other hand we keep the code (C, K) fixed, and we take into consideration that the 
parity check bits are unbiased, we get a minimum value at 6 7^ |. 



Appendix B. Cavity method 

We can derive mean field equations for a specific graph instance using the cavity method 
|41] . The cavity method gives us a link between the different mean field solutions we 
find using the replica method and different decoding algorithms. First, we derive the 
cavity equations for a typical solution cr with a weight given by (ITT]) . We define the 
cavity graph Qm^q as a graph having M spins connected to C hyperedges and q cavity 
spins connected to C — 1 hyperedges. On this graph we consider the graph operations 
defined in [H] : site addition, link addition and site iteration. We will try to count how 
the number of solutions N{e) to the equations, 

M n^-H =1' J = l,2,---,M-iV, (B.l) 
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tyO.2- 




Figure Al. The Shannon hmit of the noise g as a function of the bias &, given the rate 
R or the fraction R/h{h). The curves symmetric with respect to 6 = ^ correspond to 
a BSC channel. The other curves represent a Z-channel. The solid lines are calculated 
for a biased source and the dashed lines for a biased source with unbiased redundant 
bits. 



change when performing the 



corresponding with an energy density e — ^ 
aforementioned graph operations. If C is the set of solutions of (IB.ll) . we define 



U{e) = # <^ 1/ e C : 



E{u) 



exp (Ms(e)) 



(B.2) 



M M 

s(e) is the entropy and hi are the quenched fields. Suppose we are only interested in 
small fluctuations around some reference energy i^rcf- The probability P{E) that a 
configuration has an energy E is then given by 



P{E) ~ exp Ms 



M 



exp {(3^E) 



(B.3) 



with /3 = We define through a Legendre transform the free energy F 

= M(/3e-s(e)) . (B.4) 

We use the notation SM,q for the entropy density on the graph QM,q- Site addition is the 
graph operation which adds a site to QM,q connecting it with C hyperedges to C{K — 1) 
cavity spins. Under site addition A/'(e), we assume that M {e) fullfills 

/ X ( E 

exp (M + 1) sa/+i,o 



M+ 1 



I PiS(AE)exp [msu ,C{K-1) 



E-AE 
M 



dAE 



exp ( MsM,ciK-i) ( ) / rfAEPiS(AE)exp(-/3AE) , 



(B.5) 



where P^^^^{AE) is the distribution of energy changes under site addition. From fIB.SP 
we have 

exp [{M + l)sA/+i,o(e) - MsM,c{K-i)ie) - (3e\ 
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(B.6) 



(B.7) 



= exp log(^y"dAPi£(A£;)exp(-/3AE) 

The free energy change under site addition, AF'^^^ is thus equal to 

AF(i) = - ^ log (1 d/\EP^2 i^E) exp {-f3AE)^ . 

Link addition is the graph operation which adds a hyperedge between K cavity spins. 
Under this operation we find for Af (e) 

exp(MsM,o(e)) = exp {Msm,k{^)) dAEP[^l{AE) exp {-PAE)^ . 

(B.8) 

The free energy change under link addition AF^^^ becomes 

AF(^^) = - i log (1 dAEP}f2 i^E) exp {-PAE)^ . (B.9) 

When we start from a graph Qm,ck{k-i)), we can perform K site additions or C{K — 1) 
link additions to get a graph without cavity spins. In the limit M —>■ oo we get 

F = AF(i) - ^I^^ZlHaF^^) . 
K 

The distributions Punk (AE) and PsUe (AE) are given by 



(B.IO) 



c 



CTo cri^i,ai,2,---,aK-i,C ^=1 



K-1 



=1 \ 1=1 

5[\{vul\5{AE) , 



5 {AE + hovo) 



(B.ll) 



J = l 



with ho the external field at the new site. Pr^\ur) is the distribution of the spins on site 
r when we go to a state with energy E. We assumed that the probabilities of the cavity 
spins are uncorrelated. It is possible to find a recursion relation for Pr^\i/j.) through 



C-1 



Po{iyo,AE)= J2 []J]JPrA''rj)\]Jsi^o;l[KAH^E + hoiyo) . (B.12) 

'^i,--,i^K \r=l 1=1 J r=l \ 1=1 J 

The joint probability of the spin z/q at the new site, and the energy E' after iteration 
Ro {i^o,E'q) is 



dEodAEo exp {f3{Eo - E^ef)) Po{^0: AEo)6 {E'^ - Eq - AEq) 

exp - i?;,^)) p(^°H^o) , (B.13) 



with 



/'C-l K-l 



C-1 



^^''"Vo) = E n n PrfH^rA n M n exp ^oUo) . (B.14) 



vi,i,---,vc-l,K-\ \r=l 



r=l 



1=1 
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We see that the Eq dependency disappears. We can parametrize the spin distributions 

Pr,i{'^r,i) as 

PrliK^) = '-^^^^§M^ (B.15) 

' ' 2cosh(/5/i[) ' ^ ' 

to get the cavity or behef propagation equations 

1 ^"^ /^'^^ \ 

xo = ho + — atanh j Y\ tanh j . (B.16) 

From ( ]B.16p we can retrieve the equations (l50ll . using 
1 ^ 

7r(x|cr, ^5 a) = — ^ ^ 5 (a; - Xa-»i) 5 (2; - Za-^i) S {a - cXi) S {a - a^) , (B.17) 

and the assumption that we have large loops in the graph. We find for the free energy 
changes AF^^) and AF^^) 



C / K-l \ / K-1 



AF(i) = log exp H E 4 ^0 H ^ ) ( E 

y I'D r=l Ui,U2,---,l^K-l \ i=l / \ 1=1 

(B.18) 

AFW = -ilog|^ 5^ s(jlui;?jexp(^pf2xiu^'^ . (B.19) 

Sometimes the cavity equations (]B.16P do not converge because there are many solutions 
to these equations and each part of the graph converges to different kind of solutions. 
The reason is that the cavity spins are not uncorrelated. In that case we assume that 
there are Ai{f) solutions to the cavity equations with free energy f, i.e. 

A^(/) = # e 5^ : 4? = /) ~ exp (ME(/)) , (B.20) 



M 

We then find through a complete analogue calculation as above the IRSB equations on 

as 

Of 



a specific graph instance. In that case /x = Ij. 
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